@kickapoo as much of an annoyance as it is, you could try changing the audio codec to NONE which may help with the buffering

Is the camera and the rasbery pi connected to the same router

I was having issues as I had my rasbery pi connected to a different router, I connected both to my main router and it works fine